Fault-Tolerance Based Metrics for Evaluating System Performance in Multi-Robot Teams

The failure-prone complex operating environment of a standard multi-robot application dictates some amount of fault-tolerance to be incorporated into the system. Being able to identify the extent of fault-tolerance in a system would be a useful analysis tool for the designer. Unfortunately, it is difficult to quantify system fault-tolerance on its own. A more tangible metric for evaluation is the “effectiveness” [8] measure of fault-tolerance. Effectiveness is measured by identifying the influence of fault-tolerance towards overall system performance. In this paper, we explore the significance of the relationship between fault-tolerance and system performance, and develop metrics to measure fault-tolerance within the context of system performance. A main focus of our approach is to capture the effect of intelligence, reasoning, or learning on the effective fault-tolerance of the system, rather than relying purely on measures of redundancy. The developed metrics are designed to be application independent and can be used to evaluate and/or compare different fault-diagnostic architectures. We show the utility of the designed metrics by applying them to a sample complex heterogeneous multi-robot team application and evaluating the effective fault-tolerance exhibited by the system.